Repository navigation
feat(gateway): integrate the Wave 5 gateway features - #60
Conversation
Behind BANDIT_MODE (off, shadow or recommendation; default off), build task/model cells from stored evaluation results and nonce-protected administrator feedback, gate them on sample count, decay and the evaluation noise floor, and propose at most a one-position move in an automatic route's eligible order. Neither mode calls a provider or writes route policy; a proposal is applied only through the reviewed configuration snapshot apply with its revision check. Unknown cost or quality never ranks as favourable.
Behind CONTEXT_PAGING_ENABLED (default off), a managed client that declares the multillm_context_retrieve capability can have complete older exchanges replaced by signed retrieval handles instead of being summarized or truncated. Pages keep the exact stored JSON, never split a tool call from its results, and are scoped to the principal, session and policy revision. Bodies go in the existing R2 bucket and metadata in D1 (migration 0024), with 64 KiB per page, 1 MiB per session and a one-hour TTL. Zero-content retention disables paging, and a request whose protected content still does not fit returns 413.
Register ollama, vllm, lmstudio, llamacpp and sglang as OpenAI-compatible providers only when their base URL variable is set (all unset by default). URLs are validated to an explicit /v1 endpoint, plain HTTP is limited to loopback and private addresses, and an optional provider key is used when configured; an explicitly configured local server may be called without one. No port scanning, discovery or downloads; catalog entries carry declared capabilities with provenance.
Send content-free request observations to explicitly configured Langfuse and Helicone collectors when OBSERVABILITY_EXPORTERS_JSON lists them. Delivery uses a bounded in-memory queue with retries limited to transport failures, 429 and 5xx, and counts acknowledgements per exporter. Credentials come from variables ending in _EXPORT_CREDENTIAL. With the setting unset, nothing is sent and OTLP export is unchanged.
Score managed requests against bounded prompt injection and jailbreak rules when PROMPT_INJECTION_MODE is log or block. Block mode refuses with 422 before dispatch; both modes report the decision in X-MultiLLM-Injection-Action and a content-free log line. Key and route policies can only tighten the operator policy, and request bodies cannot change it. With the mode unset, requests are not inspected.
Behind CANARY_TRAFFIC_ENABLED, an auto route can carry a reviewed canary policy with fixed baseline and candidate weights. Sessions are assigned to a sticky cohort by an HMAC of the authenticated principal, session, route and salt revision; requests without a session stay on the baseline. Shadow mode records the proposed order without changing dispatch, and live mode puts approved candidates first for the candidate cohort while every existing filter still applies. Responses on opted-in routes carry X-MultiLLM-Canary-Cohort. Canary policies are stored in a new canary_traffic table (migration 0025).
Behind POOL_RESET_SCHEDULING_ENABLED, account pools use fresh reset observations from usage snapshots and response metadata the gateway already receives. Within the existing pool and priority tier, freshly exhausted accounts are skipped and the earliest fresh reset goes first; stale, missing or contradictory observations keep the existing order. When every account is cooling or exhausted, the existing 429 carries a Retry-After from the soonest fresh reset instead of sending another request. NanoGPT key validation keeps its existing cadence, and the scheduler adds no probes. Observations live in process for POOL_OBSERVATION_TTL_SECONDS and store only HMAC credential identities.
Behind SEMANTIC_CACHE_ENABLED and an explicit route or key opt-in in SEMANTIC_CACHE_POLICY_JSON, managed nonstream chat requests without tools can reuse a complete answer whose last user message has cosine similarity of at least 0.98. Every other part of the request, the model, sampling settings and policy revisions must match exactly, and a hit is refused when numbers, dates, negations or quoted text differ. Embeddings use an explicitly configured, priced model capped at 1,024 tokens and $0.001 per lookup. Entries are scoped to the principal, kept for 300 seconds, 256 per principal and 1 MiB each, in D1 (migration 0026) and the existing media bucket. Hits report semantic-hit provenance with no provider call; zero-content retention bypasses the cache.
OpenAI-compatible /v1/files and /v1/batches backed by D1 and R2, executed item by item through the managed chat route by the scheduled handler. Explicit spillover turns a chat request with X-MultiLLM-Priority: batch and Prefer: respond-async into a one-item batch. Both flags default to off.
Opted-in routes and key scopes replace detected emails, phones and cards with request-local HMAC placeholders before dispatch and restore them in JSON and SSE responses. Maps live only in request memory, redacted requests bypass caches and idempotency replay, and both settings default to off.
With HOSTED_RESPONSES_ENABLED on, a managed /v1/responses request with gateway_state and store keeps the completed response under a gateway-issued ID. previous_response_id continues from it in order, and GET and DELETE act only on the caller's own IDs. State lives in D1 and R2 with a 24-hour TTL, and only classified complete successes are stored.
Registers the post-authentication hooks in one named order (retention, injection, spillover, PII, hosted responses, deadline, idempotency, admission), mounts the batch and context-page routes, and connects context paging, semantic caching, canary metrics, pool usage windows, native injection, observability flushing, roleplay paging and PII scope, private storage domains and semantic cache cleanup. Batch items now run the route's response finalizers and refuse context paging before dispatch. Adds the README lines for the Wave 5 features.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| 🔵 In progress View logs |
multillm-proxy | 3fee165 | Oct 09 2026, 10:40 AM |
…o feat/observability-exports # Conflicts: # package.json
…o feat/prompt-injection # Conflicts: # package.json
… integrate/gateway-waves-1-5 # Conflicts: # package.json
…grate/gateway-waves-1-5 # Conflicts: # package.json
feat(gateway): wire the Wave 7 enterprise features together
feat(gateway): integrate the Wave 6 gateway features
efb3012
into
integrate/gateway-waves-1-4
Summary
This branch merges all eleven Wave 5 feature branches on top of the Wave 4 integration branch (#53) and wires them together. It is based on
integrate/gateway-waves-1-4, so this diff shows only Wave 5.HOSTED_RESPONSES_ENABLED, migration 0028GATEWAY_BATCHES_ENABLED,BATCH_SPILLOVER_ENABLED, migration 0027OBSERVABILITY_EXPORTERS_JSONCONTEXT_PAGING_ENABLED, migration 0024SEMANTIC_CACHE_ENABLED,SEMANTIC_CACHE_POLICY_JSON, migration 0026CANARY_TRAFFIC_ENABLED, migration 0025BANDIT_MODEOLLAMA_BASE_URLand four othersPII_REDACTION_ENABLED,PII_REDACTION_POLICY_JSONPROMPT_INJECTION_MODE,PROMPT_INJECTION_THRESHOLDPOOL_RESET_SCHEDULING_ENABLEDThe five migration PRs form one stack (#50 → #54 → #56 → #57 → #59) so each keeps its numbering contiguous. All eleven are merged here.
Integration changes
Flask hook order
register_gateway_extensionsnow runs the post-authentication hooks in one named order. Registrars no longer insert themselves by list position.X-MultiLLM-Priority: batch+Prefer: respond-asyncrequest becomes a real one-item batch (202), under the caller's normal retention.retention_conflict.retention_conflictinstead of storing content.app.pyalso registers the batch and context-page routes. The private context-page store is reached at/v1/managed-state/context-pages(2 MiB bound).Managed dispatch
capabilitiesfield is removed before the provider call.multillm_context_retrieveis an MCP tool, listed only while paging is on. It isn't in the default tool grants, so with tool grants on it needs an explicit grant.logmode and canary cohorts bypass semantic reuse.routes/unified.pyis down to 990 lines.Worker
Native direct generations:
ctx.waitUntil;Native direct routes can't resolve auto routes, so canary selection stays in the managed path.
Private domains:
semantic-cachein the control-state dispatcher, which returns 404 while its flag is off and has a 2 MiB limit;context-pagesin the managed-state dispatcher;Roleplay:
no-store.Both scope values travel in internal headers on the Durable Object hop only. The edge builds that request from an allowlist of client headers, so a caller can't supply them.
Roleplay page retrieval:
GET /v1/context/pages/<id>?roleplay_session_id=…is served in the Worker with the roleplay token's own session authority.Scheduled maintenance: adds bounded semantic cache cleanup. With every flag off, the scheduled handler registers no extra work.
Coordinator fixes
after_requesthandlers never ran for it. Those handlers cover admission release, idempotency, the hosted Responses commit and PII restoration. The item now goes throughapp.process_response, as the route itself would.tests/test_gateway_batches.pycovers this, and the test fails without the fix.context_paging_unsupportedbefore dispatch, at no cost. Before, it failed later with an authority error that was counted as an unknown outcome. Both docs say so.Known limits
context_pagesmetadata; complete JSON responses do.POST /v1/batches, doesn't check the caller's account budget at creation. Each item checks key controls, budget and admission before dispatch.cloudflare-worker.mjs(2,170 lines),routes/proxy.py(1,019) andservices/proxy_service.py(3,581) received only small call sites.Before merging to main
Five D1 migrations are included:
0024_context_pages.sqlthrough0028_responses_state.sql. A push to main does not apply them. Apply them first:Tests
tests/test_wave5_registrar.py: hook order, routes and storage plumbing;tests/test_wave5_managed.py: the managed pipeline through the registered routes;tests/test_wave5_worker.mjs: native order, private domains, roleplay and scheduled work.npm run test:worker: 1,637 passed.npm run test:integration: 7 passed.cf:migrations:checkreports 28 contiguous, additive migrations.test:deploy, free routes, client headers and thewrangler deploy --dry-runbundle all pass.tests/test_frontend_build.mjsfails locally because@tailwindcss/cliis not installed. It fails the same way on main.CI notes
python/nodejobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.AdmissionCoordinatorDurable Object migration, whichwrangler versions uploadcannot apply. This is expected untilwrangler deployruns on main.